A digital color image usually consists of an array of pixel values representing the intensity of the image at each point on a regular grid. Typically, three colors are used to generate the image. At each point on the grid the intensity of each of these colors is specified, thereby specifying both the intensity and color of the image at that grid point.
Conventional color photography records the relevant image data by utilizing three overlapping color sensing layers having sensitivities in different regions of the spectrum (usually red, green, and blue). Digital cameras, in contrast, typically utilize one array of sensors in a single “layer”.
When only one sensor array is used to detect color images, only one color may be detected at any given sensor location. As a result, these sensors do not produce a color image in the traditional sense, but rather a collection of individual color samples, which depend upon the assignment of color filters to individual sensors. This assignment is referred to as the color filter array (CFA) or the color mosaic pattern. To produce a true color image, with a full set of color samples (usually red, green and blue) at each sampling location, a substantial amount of computation is required to estimate the missing information, since only a single color was originally sensed at each location in the array.
There are a wide variety of approaches to the reconstruction problem, which is frequently referred to as the “demosaicing” task. The performance of all such algorithms depends upon the nature of the underlying CFA pattern. One such pattern is taught in U.S. Pat. No. 3,971,065. This pattern is generated by a repeating 2×2 kernel, containing two green sensors, one red sensor and one blue sensor, as illustrated below:
CFAGBGBGBGBKernelGBRGRGRGRGBRGBGBGBGBRGRGRGRGGBGBGBGBRGRGRGRG
It should be noted that this pattern has twice as many sensors in the green region of the spectrum as it does in the blue or red regions of the spectrum. The proponents of this pattern justify this choice on the grounds that the Human Visual System (HVS) is more sensitive to green. In addition, the proponents of this scheme point to the fact that it leads to relatively simple demosaicing algorithms. Specifically, if one color channel is sampled more densely than the rest, it can be interpolated most easily and then used to guide the interpolation of the less densely sampled color channels.
The argument that the HVS is more sensitive to the green portion of the spectrum is fundamentally flawed when applied to digital camera systems. The reason the human eye can afford to have more green cones than red cones, and many fewer blue cones, is that the eye's optics effectively low-pass filter the red and blue parts of the spectrum more severely than the green. Hence, the eye cannot effectively utilize additional samples in the red and blue regions of the spectrum. In effect, the eye has poor spatial resolution for images in the red and blue regions of the spectrum.
Digital cameras, in contrast, do not suffer from this limitation; hence, there is no logical reason for limiting the sampling density at any wavelength. If the sampling density in any of the color channels is reduced, there will be an increased ambiguity between spatial intensity variations and color composition of the original scene. Since such ambiguities are to be avoided, sampling densities in each of the colors should be the same.
Another problem with the small 2×2 kernel described above lies in its inability to discern color and spatial variations on image components whose spatial dimensions are small compared to the kernel's size. Consider a narrow red line (less than one sensor in width) running vertically through the image. If the line is positioned such that it is over the column of sensors having only blue and green sensors, the system will have difficulty detecting the line. In fact, if neither the blue or green sensors have at least some response in the red region of the spectrum, the line cannot be detected at all. Needless to say, accurately determining the color of this line would be difficult.
Small kernels also can present problems in demosaicing scenes having textures. Consider an image feature that is on the order of the size of the kernel. Demosaicing algorithms have difficulty determining whether the resultant sensor pattern corresponds to a smooth area in the scene with saturated color or a textured region having rapidly varying intensity of a more neutral color.
Hence, the size of the kernel should be relatively large, to assure that scene textures do not give the appearance of color variations. It should be noted, however, that there is a limit on the size of the kernel. As the kernel size grows, the implementation cost of the demosaicing algorithm typically increases. Hence, a tradeoff between cost and image quality is generally implied.
It should also be noted that images of interest to humans often include linear edges as well as lines. Hence, the sensor pattern should be designed so as to ensure that any linear feature in the image always passes through sensors of every color type. Furthermore, the distance along the linear feature between color sensors of the same type should be as small as possible. Failure to satisfy this requirement results in color ambiguity at the edges of objects in the image.
The 2×2 pattern described above is clearly a poor choice in light of these requirements. First, it has a very small kernel. Second, it undersamples two of the color channels. Third, the red and blue sensors are never adjacent to sensors of the same color. Finally, horizontal and vertical linear features never pass through all three of the sensor types.
One prior art solution to the above problems utilizes large kernels containing pseudo-random distributions of the different sensor colors to minimize the likelihood that textures will induce colors in the demosaiced image. However, such pseudo-random patterns introduce other problems. In addition to increasing the computational costs of demosaicing the image, there are regions in which there is too great a distance between color sensors of the same color in a region of the image. This can make it difficult for the demosaicing algorithm to distinguish between spatial intensity variations and color variations when it attempts to reconstruct a full color image.
Broadly, it is the object of the present invention to provide an improved color image sensor array.
It is a further object of the present invention to provide an image sensor based on a kernel that is larger than the 2×2 kernel described above.
It is yet another object of the present invention to provide an image sensor in which linear features pass through sensors for at least three different colors, whose spectral responses are linearly independent.
It is a still further object of the present invention to provide an image sensor in which color sensors for the same color are located adjacent to one another to minimize ambiguities introduced by rapidly changing intensity patterns in the image.
These and other objects of the present invention will become apparent to those skilled in the art from the following detailed description of the invention and the accompanying drawings.